Image arrangement determination apparatus, display controlling apparatus, image arrangement determination method, display controlling method, and program

ABSTRACT

An image arrangement determination apparatus, a display controlling apparatus, an image arrangement determination method, a display controlling method, and a program that can determine appropriate arrangement of an individual frame image with respect to a synthetic frame image according to a given standard are provided. A captured image acquisition section ( 60 ) acquires a plurality of images including a first image and a second image. A margin specification section ( 62 ) specifies a margin when the first image is arranged by a given first arrangement method in a region occupied by a third image that is a delivery target. An arrangement determination section ( 64 ) determines an arrangement method by which the second image is to be arranged in the margin as a second arrangement method based on a shape of the margin and a shape of the second image.

TECHNICAL FIELD

The present invention relates to an image arrangement determinationapparatus, a display controlling apparatus, an image arrangementdetermination method, a display controlling method, and a program.

BACKGROUND ART

In recent years, video delivery via the Internet has become popular. Asan example of the technology for such video delivery, a technology oftransmitting a plurality of videos collectively as one video streamexists. For example, PTL 1 and PTL 2 describe a technology of decodingvideo contents to be synthesized once, changing the size and encodingthe resulting contents after image synthesis to generate an encodedstream of a transmission target including a plurality of video contents.

CITATION LIST Patent Literature

[PTL 1]

Japanese Patent Laid-Open No. 2012-099890

[PTL 2]

Japanese Patent Laid-Open No. 2013-172374

SUMMARY Technical Problem

When a video stream including a plurality of video contents istransmitted, if it can be transmitted using existing video deliveryfacilities without introducing video delivery facilities for exclusiveuse, then this does not take time and effort for introduction of videodelivery facilities and can suppress an extra cost to be incurred.

Therefore, the inventors of the present invention are consideringtransmitting a video stream that has a plurality of frame images ofdifferent video contents arranged therein and includes frame imageshaving vertical and horizontal lengths according to standards with whichexisting video delivery facilities are compatible.

In the following description, a frame image that configures a videostream to be transmitted is referred to as synthetic frame image, andeach of frame images of respective video contents arranged in asynthetic frame image is hereinafter referred to as individual frameimage.

Here, if a plurality of individual frame images are not arrangedappropriately with respect to a synthetic frame image in compliance witha given standard, then the area occupied by a margin that does notcontribute to a video displayed in the synthetic frame image becomesgreat. However, the technology disclosed in PTL 1 or PTL 2 cannotdetermine appropriate arrangement of an individual frame image withrespect to a synthetic frame image.

The present invention has been made taking the above-described subjectinto consideration, and one of objects of the present invention residesin provision of an image arrangement determination apparatus, a displaycontrolling apparatus, an image arrangement determination method, adisplay controlling method, and a program that can determine appropriatearrangement of an individual frame image with respect to a syntheticframe image according to a given standard.

Solution to Problem

In order to solve the subject described above, an image arrangementdetermination apparatus according to the present invention includes anacquisition section configured to acquire a plurality of imagesincluding a first image and a second image, a margin specificationsection configured to specify a margin when the first image is arrangedby a given first arrangement method in a region occupied by a thirdimage that is a delivery target, and a determination section configuredto determine an arrangement method by which the second image is to bearranged in the margin as a second arrangement method based on a shapeof the margin and a shape of the second image.

In one aspect of the present invention, the image arrangementdetermination apparatus further includes a transmission sectionconfigured to transmit the third image in which the first image isarranged by the first arrangement method and the second image isarranged by the second arrangement method.

Further, in one aspect of the present invention, in a case in which itis decided that, when the second image is arranged lined up with thefirst image in a same direction as that of the first image in themargin, the second image protrudes from the margin, the determinationsection determines a method by which the second image is arranged in adirection different from that of the first image in the margin as thesecond arrangement method.

As an alternative, in a case in which it is decided that it is possibleto arrange the second image in a direction different from that of thefirst image in the margin, the determination section determines a methodof the arrangement as the second arrangement method.

Further, in one aspect of the present invention, in a case in which itis decided that it is possible to arrange the second image in the marginby changing a direction in which divisional images when the second imageis divided are lined up, the determination section determines a methodof the arrangement as the second arrangement method.

Further, in one aspect of the present invention, in a case in which themargin has a size smaller than a size of the second image, thedetermination section determines an arrangement method by which thesecond image is reduced and then arranged in the margin as the secondarrangement method.

Meanwhile, a display controlling apparatus according to the presentinvention includes an image reception section configured to receive athird image that includes a first region occupied by a first image and asecond region occupied by a second image, a data reception sectionconfigured to receive data indicative of a first arrangement method thatis an arrangement method of the first image in the third image and asecond image arrangement method that is an arrangement method of thesecond image in the third image, an extraction section configured toextract the first image and the second image from the third image basedon the first arrangement method and the second arrangement methodindicated by the data, and a display controlling section configured tocause at least one of the first image and the second image extracted tobe displayed.

Further, an image arrangement determination method according to thepresent invention includes a step of acquiring a plurality of imagesincluding a first image and a second image, a step of specifying amargin when the first image is arranged by a given first arrangementmethod in a region occupied by a third image that is a delivery target,and a step of determining an arrangement method by which the secondimage is to be arranged in the margin as a second arrangement methodbased on a shape of the margin and a shape of the second image.

Further, a display controlling method according to the present inventionincludes a step of receiving a third image that includes a first regionoccupied by a first image and a second region occupied by a secondimage, a step of receiving data indicative of a first arrangement methodthat is an arrangement method of the first image in the third image anda second image arrangement method that is an arrangement method of thesecond image in the third image, a step of extracting the first imageand the second image from the third image based on the first arrangementmethod and the second arrangement method indicated by the data, and astep of causing at least one of the first image and the second imageextracted to be displayed.

Further, a program according to the present invention causes a computerto execute a procedure for acquiring a plurality of images including afirst image and a second image, a procedure for specifying a margin whenthe first image is arranged by a given first arrangement method in aregion occupied by a third image that is a delivery target, and aprocedure for determining an arrangement method by which the secondimage is to be arranged in the margin as a second arrangement methodbased on a shape of the margin and a shape of the second image.

Further, another program according to the present invention causes acomputer to execute a procedure for receiving a third image thatincludes a first region occupied by a first image and a second regionoccupied by a second image, a procedure for receiving data indicative ofa first arrangement method that is an arrangement method of the firstimage in the third image and a second image arrangement method that isan arrangement method of the second image in the third image, aprocedure for extracting the first image and the second image from thethird image based on the first arrangement method and the secondarrangement method indicated by the data, and a procedure for causing atleast one of the first image and the second image extracted to bedisplayed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view depicting an example of a general configuration of avideo delivery system according to an embodiment of the presentinvention.

FIG. 2 is a view depicting an example of a configuration of a videodelivery apparatus according to an embodiment of the present invention.

FIG. 3 is a view depicting an example of a configuration of a usersystem according to an embodiment of the present invention.

FIG. 4 is a view depicting an example of a configuration of a headmounted display according to an embodiment of the present invention.

FIG. 5A is a view depicting an example of a texture image.

FIG. 5B is a view depicting an example of a texture image.

FIG. 6 is a view depicting an example of a virtual space.

FIG. 7A is a view depicting an example of an arrangement method of animage.

FIG. 7B is a view depicting an example of an arrangement method of animage.

FIG. 8 is a functional block diagram depicting an example of functionsincorporated in a video delivery apparatus and a head mounted displayserver according to an embodiment of the present invention.

FIG. 9A is a view depicting an example of a manifest file.

FIG. 9B is a view depicting an example of a manifest file.

FIG. 10A is a flow chart depicting an example of a flow of processingperformed by a video delivery apparatus according to an embodiment ofthe present invention.

FIG. 10B is a flow chart depicting an example of a flow of processingperformed by a video delivery apparatus according to an embodiment ofthe present invention.

FIG. 10C is a flow chart depicting an example of a flow of processingperformed by a video delivery apparatus according to an embodiment ofthe present invention.

FIG. 11 is a view depicting an example of a region corresponding to amargin when a region corresponding to a virtual reality (VR) frame imageis secured in an upper left corner of a region corresponding to asynthetic frame image.

FIG. 12 is a view depicting an example of a region corresponding to avirtual screen frame image.

FIG. 13 is a flow chart depicting an example of a flow of processingperformed in a head mounted display according to an embodiment of thepresent invention.

FIG. 14 is a view depicting an example of an arrangement method of animage.

FIG. 15 is a view depicting an example of a manifest file.

FIG. 16 is a view depicting an example of an arrangement method of animage.

FIG. 17 is a view depicting an example of a manifest file.

FIG. 18 is a view depicting an example of an arrangement method of animage.

FIG. 19 is a view depicting an example of a manifest file.

DESCRIPTION OF EMBODIMENT

In the following, an embodiment of the present invention is described indetail with reference to the drawings.

FIG. 1 is a view depicting an example of a general configuration of avideo delivery system 10 according to an embodiment of the presentinvention. As depicted in FIG. 1, the video delivery system 10 accordingto the present embodiment includes a video delivery apparatus 12, a usersystem 14 and cameras 16 (a camera 16 a and another camera 16 b). Thevideo delivery apparatus 12 and the user system 14 are connected to acomputer network 18 such as the Internet such that the video deliveryapparatus 12 and the user system 14 can communicate with each other.Further, the cameras 16 and the video delivery apparatus 12 arecommunicatable with each other, and a video captured by any of thecameras 16 is transmitted to the video delivery apparatus 12.

FIG. 2 is a view depicting an example of a configuration of the videodelivery apparatus 12 according to the present embodiment. The videodelivery apparatus 12 according to the present embodiment is a computersuch as, for example, a video delivery server. As depicted in FIG. 2,the video delivery apparatus 12 according to the present embodimentincludes, for example, a processor 12 a, a storage section 12 b, acommunication section 12 c, an encoder section 12 d, a display section12 e, an operation section 12 f and so forth.

The processor 12 a is a control device such as, for example, amicroprocessor that operates in accordance with a program installed inthe video delivery apparatus 12.

The storage section 12 b is a storage device such as, for example, aread only memory (ROM) or a random access memory (RAM), a hard diskdrive or the like. Into the storage section 12 b, a program to beexecuted by the processor 12 a and so forth are stored.

The communication section 12 c is a communication interface such as, forexample, a network board or a wireless local area network (LAN) module.

The encoder section 12 d is a hardware encoder that encodes, forexample, data of a content of a video or the like generated by theprocessor 12 a or data of a content of a video or the like stored in thestorage section 12 b.

The display section 12 e is a display device such as a liquid crystaldisplay and displays various images in accordance with an instructionfrom the processor 12 a.

The operation section 12 f is a user interface such as a keyboard, amouse, a controller of a game console or the like, and accepts anoperation input made by a user and outputs a signal representative ofthe substance of the operation input to the processor 12 a.

FIG. 3 is a view depicting an example of a configuration of the usersystem 14 according to an embodiment of the present invention. FIG. 4 isa view depicting an example of a configuration of a head mounted display(HMD) 20 according to the present embodiment.

As depicted in FIG. 3, the user system 14 according to the presentembodiment includes the HMD 20, an entertainment apparatus 22, a relayapparatus 24, a display 26, a camera microphone unit 28, and acontroller 30.

The HMD 20 according to the present embodiment includes, for example, asdepicted in FIG. 4, a processor 20 a, a storage section 20 b, acommunication section 20 c, an inputting and outputting section 20 d, adecoder section 20 e, a display section 20 f, a sensor section 20 g anda sound outputting section 20 h.

The processor 20 a is a program-controlled device such as amicroprocessor that operates in accordance with a program, for example,installed in the HMD 20.

The storage section 20 b is a storage device such as, for example, a ROMor a RAM. In the storage section 20 b, a program to be executed by theprocessor 20 a and so forth are stored.

The communication section 20 c is a communication interface such as, forexample, a network board or a wireless LAN module.

The inputting and outputting section 20 d is an input/output port suchas, for example, a high-definition multimedia interface (HDMI)(registered trademark) port, a universal serial bus (USB) port, or anauxiliary (AUX) port.

The decoder section 20 e is hardware for performing decoding of data,for example, of a content. The decoder section 20 e decodes data of acontent received, for example, by the communication section 20 c or theinputting and outputting section 20 d.

The display section 20 f is a display such as, for example, a liquidcrystal display or an organic electroluminescence (EL) display disposedon the front side of the HMD 20, and displays a video generated by theentertainment apparatus 22, a video received by the entertainmentapparatus 22, or the like. Further, the display section 20 f isaccommodated in a housing of the HMD 20. The display section 20 f mayoutput a video represented by a video signal outputted from theentertainment apparatus 22, relayed by the relay apparatus 24, andreceived, for example, by the communication section 20 c or theinputting and outputting section 20 d. The display section 20 faccording to the present embodiment can display a three-dimensionalimage, for example, by displaying an image for the left eye and an imagefor the right eye. It is to be noted that the display section 20 f mayotherwise be of the type that cannot display a three-dimensional imagebut can display only a two-dimensional image.

The sensor section 20 g includes sensors such as, for example, anacceleration sensor and a motion sensor. The sensor section 20 g outputsmeasurement results of a rotation amount and an amount of movement ofthe HMD 20 at a predetermined sampling rate to the processor 20 a.

The sound outputting section 20 h is, for example, a headphone, aspeaker, and so forth and outputs sound represented by sound datagenerated by the entertainment apparatus 22, sound represented by sounddata received by the entertainment apparatus 22, and so forth. The soundoutputting section 20 h outputs sound represented by a sound signaloutputted from the entertainment apparatus 22, relayed by the relayapparatus 24, and received, for example, by the communication section 20c or the inputting and outputting section 20 d.

The entertainment apparatus 22 according to the present embodiment is acomputer such as, for example, a game console, a digital versatile disc(DVD) player, or a Blu-ray (registered trademark) player. Theentertainment apparatus 22 according to the present embodiment generatesa video or a sound, for example, by execution of a game program storedor by reproduction of a content recorded on an optical disk or the like.Then, the entertainment apparatus 22 according to the present embodimentoutputs a video signal representative of a video generated or an audiosignal representative of a sound generated to the HMD 20 or the display26 through the relay apparatus 24.

Further, the entertainment apparatus 22 according to the presentembodiment receives a video signal or an audio signal transmitted fromthe video delivery apparatus 12 and outputs the video signal or theaudio signal to the HMD 20 or the display 26 through the relay apparatus24.

The relay apparatus 24 according to the present embodiment is a computerthat relays and outputs a video signal or an audio signal outputted fromthe entertainment apparatus 22 to the HMD 20 or the display 26.

The camera microphone unit 28 according to the present embodimentincludes a camera 28 a that outputs, for example, an image obtained bycapturing an imaging object to the entertainment apparatus 22 and amicrophone 28 b that acquires surrounding sound, converts the sound intoaudio data, and outputs the audio data to the entertainment apparatus22. Further, the camera 28 a according to the present embodiment is astereo camera.

The HMD 20 and the relay apparatus 24 are connected to each other, forexample, by an HDMI cable or a USB cable such that they can transferdata therebetween. It is to be noted that the HMD 20 may performtransfer of data to and from the relay apparatus 24 through thecommunication section 20 c. The entertainment apparatus 22 and the relayapparatus 24 are connected to each other, for example, by an HDMI cable,a USB cable, or the like such that they can transfer data therebetween.The relay apparatus 24 and the display 26 are connected to each other,for example, by an HDMI cable or the like. The entertainment apparatus22 and the camera microphone unit 28 are connected to each other, forexample, an AUX cable or the like.

The controller 30 according to the present embodiment is an operationinputting apparatus for performing operation inputting to theentertainment apparatus 22. The user can perform various kinds ofoperation inputting using the controller 30 by depressing a directionkey or a button or tilting an operation stick provided on the controller30. The controller 30 according to the present embodiment outputs inputdata associated with operation inputting to the entertainment apparatus22. Further, the controller 30 according to the present embodimentincludes a USB port. The controller 30 can output input data by wire tothe entertainment apparatus 22 by connecting to the entertainmentapparatus 22 with a USB cable. Further, the controller 30 according tothe present embodiment includes a wireless communication module or thelike such that it can also output input data wirelessly to theentertainment apparatus 22.

In the present embodiment, a video representative of a manner of anevent venue where an event such as a concert takes place is captured bythe cameras 16. Here, it is assumed that, for example, videos differentfrom each other are captured from positions different from each other bythe camera 16 a and the camera 16 b. For example, a general state of theevent venue may be imaged by the camera 16 a and a performer on a stagein the event venue may be imaged particularly by the camera 16 b.Further, the camera 16 a is, for example, an omnidirectional camera andmay be a camera that can capture a video of a viewing angle greater than180 degrees.

In the following description, a video captured by the camera 16 a isreferred to as VR video, and a video captured by the camera 16 b isreferred to as virtual screen video.

Thus, the video delivery apparatus 12 according to the presentembodiment receives a video captured, for example, by the camera 16 aand/or the camera 16 b. Then, the video delivery apparatus 12 generatesa frame image in which, for example, a frame image captured by thecamera 16 a and a frame image captured by the camera 16 b at the sametime as the frame image captured by the camera 16 a are arranged. Here,for example, a frame image in which a frame image captured by the camera16 a and a frame image captured by the camera 16 b, which are associatedwith a same timestamp, are arranged may be generated.

In the following description, a frame image captured by the camera 16 aand a frame image captured by the camera 16 b are each referred to asindividual frame image, and a frame image in which a plurality ofindividual frame images are arranged is referred to as synthetic frameimage. Further, an individual frame image captured by the camera 16 a isreferred to as VR frame image 40, and an individual frame image capturedby the camera 16 b is referred to as virtual screen frame image 42(refer to FIGS. 5A and 5B).

Then, in the present embodiment, after a video including a syntheticframe image generated in this manner is encoded, it is delivered fromthe video delivery apparatus 12 to the user system 14. Then, the videodelivered from the video delivery apparatus 12 is decoded by the usersystem 14. Then, the user system 14 extracts, from the synthetic frameimage included in the decoded video, the VR frame image 40 and thevirtual screen frame image 42. Then, the user system 14 generatestexture images 44 (44 a and 44 b) exemplified in FIGS. 5A and 5B on thebasis of the extracted VR frame image 40 and virtual screen frame image42.

In the present embodiment, for example, each texture image 44 generatedis mapped to the inside of a background object 48 that is a sphericalvirtual object arranged in a virtual space 46 exemplified in FIG. 6.

Then, an image representative of a state when a user views a gazedirection 52 from a viewpoint 50 arranged at the center of thebackground object 48 is displayed on the display section 20 f of the HMD20. In the present embodiment, for example, part of the texture image 44mapped to the inner side of the background object 48 is displayed on thedisplay section 20 f. Here, for example, the position of the viewpoint50 and/or the gaze direction 52 in the virtual space 46 may be changedin response to a change of the position or the direction of the head ofthe user having the HMD 20 mounted thereon which change is detected bythe sensor section 20 g. This makes it possible for the user tovisualize various portions of the texture image 44 by changing thedirection of its head.

In FIG. 5A, the VR frame image 40 is indicated as the texture image 44a. Meanwhile, in FIG. 5B, an image in which the virtual screen frameimage 42 is arranged in superposition at a position at which an image ofthe stage is arranged in the VR frame image 40 is indicated as thetexture image 44 b. It is to be noted that, in the texture image 44 bdepicted in FIG. 5B, the virtual screen frame image 42 may be arrangedin superposition on the VR frame image 40 after deformed such that it isdisplayed as a rectangular image on the display section 20 f. Further,the texture image 44 b in which the virtual screen frame image 42 isarranged in superposition on the VR frame image 40 need not be mapped tothe inner side of the background object 48. For example, the virtualscreen frame image 42 may be mapped as the texture image 44 b to aplate-shaped virtual object arranged in the virtual space 46.

Further, in the present embodiment, a state in which the VR frame image40 is used as the texture image 44 a and another state in which the VRframe image 40 on which the virtual screen frame image 42 is superposedis used as the texture image 44 b can be switched therebetween. Here, itmay be made possible to switch the two states just described, forexample, by the user operating the controller 30.

In this manner, according to the present embodiment, individual frameimages captured by the plurality of cameras 16 can be providedcollectively as one synthetic frame image to the user.

Here, it is assumed that, in the present embodiment, the vertical andhorizontal lengths of a synthetic frame image are determined in advance.Then, in the present embodiment, an arrangement method for the virtualscreen frame image 42 is determined in the following manner on the basisof the shape of a margin when the VR frame image 40 is arranged by anarrangement method determined in advance on a synthetic frame image.Therefore, according to the present embodiment, appropriate arrangementof individual frame images in a synthetic frame image according to agiven standard can be determined.

In FIGS. 7A and 7B, examples of the arrangement method of the VR frameimage 40 and the virtual screen frame image 42 are depicted. It isassumed that, not only in the example of FIG. 7A but also in the exampleof FIG. 7B, the VR frame image 40 is arranged in the upper left cornerof the synthetic frame image.

In the example of FIG. 7A, the virtual screen frame image 42 is arrangedon the right side of the VR frame image 40 in a state in which it isrotated by 90 degrees clockwise with respect to the VR frame image 40.Meanwhile, in the example of FIG. 7B, the virtual screen frame image 42is horizontally divided into three and is arranged under the VR frameimage 40 after the lined up direction of the divisional images of thedivided virtual screen frame image 42 is changed from the verticaldirection to the horizontal direction. As the arrangement method for thevirtual screen frame image 42, for example, a method of arranging thesame in rotation or a method of arranging the same in division may bedetermined in this manner.

In the following, functions of the video delivery apparatus 12 and theHMD 20 according to the present embodiment and processes executed by thevideo delivery apparatus 12 and the HMD 20 are described further.

FIG. 8 is a functional block diagram depicting an example of functionsincorporated in the video delivery apparatus 12 and the HMD 20 accordingto the present embodiment. It is to be noted that, in the video deliveryapparatus 12 and the HMD 20 according to the present embodiment, all ofthe functions depicted in FIG. 8 need not be incorporated, and functionsother than the functions depicted in FIG. 8 may be incorporated.

As depicted in FIG. 8, the video delivery apparatus 12 functionallyincludes, for example, a captured image acquisition section 60, a marginspecification section 62, an arrangement determination section 64, amanifest file generation section 66, a manifest file storage section 68,a manifest file transmission section 70, a video generation section 72,and a video transmission section 74. The captured image acquisitionsection 60 is incorporated principally as the processor 12 a and thecommunication section 12 c. The margin specification section 62, thearrangement determination section 64, and the manifest file generationsection 66 are incorporated principally as the processor 12 a. Themanifest file storage section 68 is incorporated principally as thestorage section 12 b. The manifest file transmission section 70 and thevideo transmission section 74 are incorporated principally as thecommunication section 12 c. The video generation section 72 isincorporated principally as the processor 12 a and the encoder section12 d. The video delivery apparatus 12 according to the presentembodiment plays a role as an image arrangement determination apparatusfor determining arrangement of the VR frame image 40 and the virtualscreen frame image 42 in a synthetic frame image.

The functions described above may be incorporated by the processor 12 aexecuting a program installed in the video delivery apparatus 12 that isa computer and including instructions corresponding to the functionsdescribed above. This program may be supplied to the video deliveryapparatus 12 through a computer-readable information storage medium suchas, for example, an optical disk, a magnetic disk, a magnetic tape, amagneto-optical disk, or a flash memory or through the Internet or thelike.

Meanwhile, as depicted in FIG. 8, the HMD 20 functionally includes, forexample, a manifest file reception section 80, a manifest file storagesection 82, a video reception section 84, an image extraction section86, a display controlling operation acceptance section 88, and a displaycontrolling section 90. The manifest file reception section 80 and thevideo reception section 84 are incorporated principally as the inputtingand outputting section 20 d or the communication section 20 c. Themanifest file storage section 82 is incorporated principally as thestorage section 20 b. The image extraction section 86 is incorporatedprincipally as the processor 20 a and the decoder section 20 e. Thedisplay controlling operation acceptance section 88 is incorporatedprincipally as the processor 20 a and the inputting and outputtingsection 20 d. It is to be noted that the display controlling operationacceptance section 88 may be incorporated principally as the processor20 a and the communication section 20 c. The display controlling section90 is incorporated principally as the processor 20 a and the displaysection 20 f. The HMD 20 according to the present embodiment plays arole as a display controlling apparatus for causing at least one of theVR frame image 40 and the virtual screen frame image 42 to be displayed.

The functions described above may be incorporated by the processor 20 aexecuting a program installed in the HMD 20 that is a computer andincluding instructions corresponding to the functions described above.This program may be supplied to the HMD 20 through a computer-readableinformation storage medium such as, for example, an optical disk, amagnetic disk, a magnetic tape, a magneto-optical disk, or a flashmemory or through the Internet or the like.

The captured image acquisition section 60 acquires a plurality of imagesincluding, in the present embodiment, for example, a VR frame image 40and a virtual screen frame image 42. The captured image acquisitionsection 60 acquires, for example, a video captured by the camera 16 aand a video captured by the camera 16 b.

The margin specification section 62 specifies a margin when, in thepresent embodiment, for example, a VR frame image 40 is arranged into aregion occupied by a synthetic frame image of a delivery target by apredetermined first arrangement method. Here, the predetermined firstarrangement is, in the examples described hereinabove, a method ofarranging the VR frame image 40 in the upper left corner of thesynthetic frame image.

Here, the margin specification section 62 may specify a margin on thebasis of the vertical and horizontal lengths of a VR frame image 40, avirtual screen frame image 42, and a synthetic frame image inputted, forexample, by an operator (person in charge of video delivery) of thevideo delivery apparatus 12. Alternatively, the margin specificationsection 62 may specify a margin, for example, on the basis of thevertical and horizontal lengths of a VR frame image 40 and a virtualscreen frame image 42 acquired by the captured image acquisition section60 and a synthetic frame image indicated by a setting file or the likeof the video delivery apparatus 12.

The arrangement determination section 64 determines, in the presentembodiment, for example, on the basis of the shape of the marginspecified by the margin specification section 62 and the shape of thevirtual screen frame image 42, a second arrangement method of arrangingthe virtual screen frame image 42 in the margin. Here, such a method ofarranging the virtual screen frame image 42 in rotation or a method ofarranging the virtual screen frame image 42 in division as describedabove may be determined as the second arrangement method. Alternatively,for example, an arrangement method of reducing the virtual screen frameimage 42 and arranging the reduced virtual screen frame image 42 into amargin as hereinafter described may be determined as the secondarrangement method.

The manifest file generation section 66 generates data that indicates,in the present embodiment, for example, at least the first arrangementmethod described hereinabove and the second arrangement method describedhereinabove and indicates values of attributes of individual frameimages. Here, for example, a manifest file in which values of attributesof individual frame images are described may be generated. Further, inthe present embodiment, the manifest file generation section 66 storesthe generated manifest file into the manifest file storage section 68.

FIG. 9A depicts an example of a manifest file associated with thearrangement of FIG. 7A, and FIG. 9B depicts an example of a manifestfile associated with the arrangement of FIG. 7B.

In the examples of FIGS. 9A and 9B, values of attributes of the VR frameimage 40 are indicated from a line next to [VR_Movie_1] to a linepreceding to [VirtualScreen_1].

For example, the form of projection of the VR frame image 40 is set as avalue of the attribute Projection. As depicted in FIGS. 9A and 9B, inthe case where the form of projection is equal distance cylindricalprojection, the value “EQ” is set as a value of the attributeProjection.

Further, for example, whether or not the VR frame image 40 is athree-dimensional image is set as a value of the attribute 3D. Here, inthe case where the VR frame image 40 is a three-dimensional image, theform of the same is set as a value of the attribute 3D. For example, inthe case where the VR frame image 40 is a two-dimensional image, thevalue “NO” is set as a value of the attribute 3D. On the other hand, inthe case where the VR frame image 40 is a three-dimensional image of theside by side type, the value SBS is set as a value of the attribute 3D.Further, for example, in the case where the VR frame image 40 is athree-dimensional image of the top and bottom type, the value “TB” isset as a value of the attribute 3D.

Further, for example, the coordinate value in the horizontal directionand the coordinate value in the vertical direction of the upper leftcorner of the VR frame image 40 in the case where the upper left cornerof a synthetic frame image is determined as the origin are set as avalue of the attribute left_position and a value of the attributetop_position, respectively. Further, for example, the horizontal lengthand the vertical length of the VR frame image 40 are set as a value ofthe attribute x_size and a value of the attribute y_size. Further, forexample, an angle in the horizontal direction and an angle in thevertical direction covered by the VR frame image 40 are set as a valueof the attribute h_angle and a value of the attribute v_angle,respectively. It is to be noted that, depending upon the value of theattribute Projection, the value of the attribute h_angle and the valueof the attribute v_angle may not be set. Further, information that isunique to the projection form and is to be set may be set as a value ofthe attribute Extend_info.

In the examples of FIGS. 9A and 9B, values of attributes of the virtualscreen frame image 42 are indicated in lines beginning with a line nextto [VirtualScreen_1].

For example, whether or not the virtual screen frame image 42 is athree-dimensional image is set as a value of the attribute 3D. Here, inthe case where the virtual screen frame image 42 is a three-dimensionalimage, the type of the same is set as a value of the attribute 3D. Forexample, in the case where the virtual screen frame image 42 is atwo-dimensional image, the value “NO” is set as a value of the attribute3D. On the other hand, for example, in the case where the virtual screenframe image 42 is a three-dimensional image of the side by side type,the value “SBS” is set as a value of the attribute 3D. Further, forexample, in the case where the virtual screen frame image 42 is athree-dimensional image of the top and bottom type, the value “TB” isset as a value of the attribute 3D.

Further, for example, whether or not the virtual screen frame image 42is to be arranged in a direction rotated with respect to the VR frameimage 40 is set as a value of the attribute virtualscreen_rotate. Forexample, in the case where the virtual screen frame image 42 is to bearranged in a direction same as that of the VR frame image 40, 0 is setas the value of the attribute virtualscreen_rotate. On the other hand,for example, in the case where the virtual screen frame image 42 is tobe arranged rotated by 90 degrees clockwise with respect to the VR frameimage 40, 1 is set as a value of the attribute virtualscreen_rotate.Further, for example, in the case where the virtual screen frame image42 is to be arranged rotated by 90 degrees counterclockwise with respectto the VR frame image 40, 2 is set as a value of the attributevirtualscreen_rotate.

Further, for example, the division number of the virtual screen frameimage 42 is set as a value of the attribute virtualscreen_devided. Here,in the case where the virtual screen frame image 42 is divided into n, nis set as the value of the attribute virtualscreen_devided. It is to benoted that, in the case where the virtual screen frame image 42 is notdivided, 1 is set as the value of the attribute virtualscreen_devided.

Further, the arrangement form of the divided virtual screen frame image42 is set as a value of the attribute virtualscreen_devided_Type. Forexample, in the case where the divisional images of the divided virtualscreen frame image 42 are arranged side by side in the horizontaldirection in a synthetic frame image, w is set as a value of theattribute virtualscreen_devided_Type. In this case, by rearranging thevirtual screen frame image 42 after division in which the divisionalimages are arranged side by side horizontally in the synthetic frameimage such that the divisional images are arranged vertically, thevirtual screen frame image 42 before division is reproduced. On theother hand, in the case where the divisional images of the dividedvirtual screen frame image 42 are arranged side by side in the verticaldirection, 2 is set as a value of the attributevirtualscreen_devided_Type. In this case, by rearranging the virtualscreen frame image 42 after division in which the divisional images arearranged side by side vertically in the synthetic frame image such thatthe divisional images are arranged horizontally, the virtual screenframe image 42 before division is reproduced.

Further, the coordinate value in the horizontal direction of the upperleft corner of the virtual screen frame image 42 in the case where, forexample, the upper left corner of the synthetic frame image is theorigin is set as a value of the attribute left_position_m (1≤m≤n).Further, the coordinate value in the vertical direction of the upperleft corner of the virtual screen frame image 42 in the case where theupper left corner of the synthetic frame image is the origin is set as avalue of the attribute top_position_m (1≤m≤n). Further, the horizontallength and the vertical length of the virtual screen frame image 42 areset as a value of the attribute x_size_m and a value of the attributey_size_m (1≤m≤n), respectively. It is to be noted that the value nindicates a division number of the virtual screen frame image 42 asdescribed hereinabove. Meanwhile, the value m indicates an order numberof a divisional image when the virtual screen frame image 42 is dividedinto n. It is to be noted that, in the case where it is determined thatthe arrangement method for reducing the virtual screen frame image 42and then arranging the virtual screen frame image 42 in a margin is tobe included into the second arrangement method, the size after thereduction may be set as a value of the attribute x_size_m and a value ofthe attribute y_size_m.

For example, it is assumed that the virtual screen frame image 42 isdivided into three as depicted in FIG. 7B. In this case, as depicted inFIG. 9B, for the first divisional image of the divided virtual screenframe image 42, a value of the attribute left_position_1, a value of theattribute top_position_1, a value of the attribute x_size_1, and a valueof the attribute y_size_1 are set. Similarly, for the second divisionalimage of the divided virtual screen frame image 42, a value of theattribute left_position_2, a value of the attribute top_positoin_2, avalue of the attribute x_size_2, and a value of the attribute y_size_2are set. Similarly, for the third divisional image of the dividedvirtual screen frame image 42, a value of the attribute left_position_3,a value of the attribute top_position_3, a value of the attributex_size_3, and a value of the attribute y_size_3 are set.

In the present example, the first divisional image, second divisionalimage and third divisional image of the divided virtual screen frameimage 42 are arranged side by side in this order from the left. Further,in this case, the original virtual screen frame image 42 is reproducedby arranging the virtual screen frame image 42 after division such thatthe divisional image arranged on the left, divisional image arranged inthe middle, and divisional image arranged on the right are arranged sideby side vertically in this order from above.

Further, the magnitude with which, for example, the virtual screen frameimage 42 is superimposed on the VR frame image 40 in the texture image44 is set as a value of the attribute overlay_size_degree_p. Then, theposition in the horizontal direction at which the virtual screen frameimage 42 is superimposed on the VR frame image 40 in the texture image44 is set as a value of the attribute overlay_size_position_x_degree_p.Further, the position in the vertical direction at which the virtualscreen frame image 42 is superimposed on the VR frame image 40 in thetexture image 44 is set as a value of the attributeoverlay_size_position_y_degree_p.

It is to be noted that the value of the attribute overlay_size_degree_p,the value of the attribute overlay_size_position_x_degree_p, and thevalue of the attribute overlay_size_position_y_degree_p may berepresented, for example, by angles. Here, the value p is a numberrepresenting an order number of the VR frame image 40. For example, thesynthetic frame image sometimes includes a plurality of VR frame images40. In this case, a value of the attribute overlay_size_degree_p, avalue of the attribute overlay_size_position_x_degree_p, and a value ofthe attribute overlay_size_position_y_degree_p corresponding to each ofthe VR frame images 40 are set.

In the examples depicted in FIGS. 7A and 7B, the synthetic frame imageincludes a single VR frame image 40. Therefore, in the manifest filesdepicted in FIGS. 9A and 9B, a value of the attributeoverlay_size_degree_1, a value of the attributeoverlay_size_position_x_degree_1, and a value of the attributeoverlay_size_position_y_degree_1 are set.

It is to be noted that the values of the attributes indicated by themanifest file may be represented by relative values. For example, as thevalue of the attribute left_position or the value of the attributex_size, a ratio of the value to the horizontal length of the syntheticframe image may be represented in terms of percentage. Further, forexample, as the value of the attribute top_position or the value of theattribute y_size, a ratio of the value to the vertical length of thesynthetic frame image may be represented in terms of percentage.

The manifest file generation section 66 may generate a manifest file inwhich values of attributes inputted by a person in charge of videodelivery are set. Further, for example, the manifest file generationsection 66 may generate a manifest file in which values of attributesare set on the basis of meta information associated with a VR frameimage 40, a virtual screen frame image 42, or a synthetic frame image.

Further, for example, the manifest file generation section 66 maygenerate a manifest file in which values of attributes according toarrangement determined by the arrangement determination section 64 areset. Further, the values of the attributes set in this manner may bechangeable, for example, by the operation section 12 f operated by aperson in charge of video delivery.

The manifest file storage section 68 stores, in the present embodiment,for example, data indicative of values of attributes of individual frameimages such as, for example, a manifest file generated by the manifestfile generation section 66.

The manifest file transmission section 70 transmits, in the presentembodiment, for example, data indicative of values of attributes ofindividual frame images such as a manifest file generated by themanifest file generation section 66. Here, for example, the manifestfile may be transmitted to the HMD 20 through the entertainmentapparatus 22 and the relay apparatus 24.

The video generation section 72 generates a third image (for example, asynthetic frame image) on the basis of, in the present embodiment, forexample, a first image (for example, a VR frame image 40) and a secondimage (for example, a virtual screen frame image 42). Here, for example,a synthetic frame image may be generated in which a VR frame image 40 isarranged by a first arrangement method indicated by the manifest fileand a virtual screen frame image 42 is arranged by a second arrangementmethod indicated by the manifest file. Here, for example, as depicted inFIGS. 7A and 7B, a synthetic frame image in which a VR frame image 40and a virtual screen frame image 42 are arranged is generated. Then, inthe present embodiment, for example, the video generation section 72encodes a plurality of synthetic frame images and/or meta information togenerate a video of a delivery target.

In the present embodiment, for example, the video transmission section74 transmits a synthetic frame image in which a VR frame image 40 isarranged by the first arrangement method and a virtual screen frameimage 42 is arranged by the second arrangement method. For example, thevideo transmission section 74 transmits an encoded video of a deliverytarget generated by the video generation section 72. Here, the video maybe transmitted to the HMD 20, for example, through the entertainmentapparatus 22 and the relay apparatus 24.

In the present embodiment, for example, the manifest file receptionsection 80 receives data indicative of values of attributes ofindividual frame images such as a manifest file transmitted by themanifest file transmission section 70. Then, the manifest file receptionsection 80 stores the data such as a manifest file indicative of valuesof attributes of individual frame image into the manifest file storagesection 82.

In the present embodiment, for example, the manifest file storagesection 82 stores the data indicative of values of attributes ofindividual frame images such as a manifest file received by the manifestfile reception section 80.

In the present embodiment, for example, the video reception section 84receives a video transmitted from the video transmission section 74.

In the present embodiment, for example, the image extraction section 86decodes a video transmitted from the video transmission section 74 andextracts a synthetic frame image. Then, in the present embodiment, forexample, the image extraction section 86 extracts a VR frame image 40and a virtual screen frame image 42 on the basis of data such as amanifest file indicative of values of attributes of individual frameimages and a synthetic frame image. Here, for example, the imageextraction section 86 may extract, on the basis of a first arrangementmethod and a second arrangement method indicated by the manifest file, aVR frame image 40 and a virtual screen frame image 42 from the syntheticframe image. Further, for example, the image extraction section 86 mayrotate the virtual screen frame image 42 after rotation arranged in thesynthetic frame image to reproduce the virtual screen frame image 42before rotation. Alternatively, for example, the direction in whichdivisional images of the virtual screen frame image 42 after divisionarranged in a synthetic frame image are lined up may be changed toreproduce the virtual screen frame image 42 before division.

The display controlling operation acceptance section 88 accepts anoperation for controlling an image displayed on the display section 20 fsuch as, for example, in the present embodiment, an operation forchanging the gaze direction 52 such as an operation of the user turningits head or an operation for switching the texture image 44.

In the present embodiment, for example, the display controlling section90 generates a texture image 44 based on an operation accepted by thedisplay controlling operation acceptance section 88 and a VR frame image40 and a virtual screen frame image 42 extracted by the image extractionsection 86. Then, the display controlling section 90 controls thedisplay section 20 f to display the generated texture image 44.

Here, the display controlling section 90 may cause at least one of anextracted VR frame image 40 and a virtual screen frame image 42 to bedisplayed.

For example, the display controlling section 90 may retain viewpointposition data indicative of the position of the viewpoint 50, gazedirection data indicative of a gaze direction 52, and a display controlflag indicative of whether or not a virtual screen frame image 42 is tobe displayed. Here, the value of the viewpoint position data, the valueof the gaze direction data, and the value of the display control flagmay be set in response to an operation accepted by the displaycontrolling operation acceptance section 88. Then, a texture image 44according to the value of the display control flag may be arranged inthe virtual space 46. Then, an image representative of a state in whicha gaze direction 52 indicated by the value of the gaze direction data isviewed from the viewpoint 50 arranged at a position indicated by thevalue of the viewpoint position data may be displayed on the displaysection 20 f. Here, for example, in the case where the value of thedisplay control flag is 0, the HMD 20 may be controlled such that thevirtual screen frame image 42 is not displayed thereon, but in the casewhere the value of the display control flag is 1, the HMD 20 may becontrolled such that the virtual screen frame image 42 is displayed.

For example, in the case where an event held in an event venue is to belive relayed, capturing of a VR frame image 40 by the camera 16 a andcapturing of a virtual screen frame image 42 by the camera 16 b may beperformed at a predetermined sampling rate. Then, generation of asynthetic frame image based on the VR frame image 40 and the virtualscreen frame image 42 may be performed in accordance with values ofattributes represented by a manifest file at the predetermined samplingrate described above. Then, a video in which the synthetic frame imagegenerated at the predetermined sampling rate in this manner and metainformation are encoded may be delivered to the user system 14. Then, avideo of the live relay of the event venue may be displayed on a realtime basis on the display section 20 f.

Here, an example of a flow of processing for determining an arrangementmethod of a virtual screen frame image 42, which is performed by thevideo delivery apparatus 12 according to the present embodiment, isdescribed with reference to flow charts exemplified in FIGS. 10A to 10C.Note that it is assumed that, in the present example of processing, thevertical and horizontal lengths of a synthetic frame image aredetermined in advance.

First, the margin specification section 62 specifies a margin when a VRframe image 40 is arranged in the upper left corner of a synthetic frameimage (S101). FIG. 11 depicts an example of a region R3 corresponding toa margin when a region R2 corresponding to the VR frame image 40 issecured in the upper left corner of a region R1 corresponding to thesynthetic frame image. Here, as depicted in FIG. 11, the horizontallength of a narrow portion in the region R3 having an inverted L shapeis represented by L1, and the vertical length of the narrow portion isrepresented by L2.

Then, the arrangement determination section 64 specifies the verticaland horizontal lengths of the virtual screen frame image 42 acquired bythe captured image acquisition section 60 (S102).

Then, the arrangement determination section 64 decides whether or notthe size of the margin specified by the process indicated at S101 isequal to or greater than the size of the virtual screen frame image 42whose vertical and horizontal lengths are specified by the processindicated at 3102 (S103).

In the case where it is decided that the size of the margin is not equalto or greater than the size of the virtual screen frame image 42 (S103:N), the arrangement determination section 64 decides that an arrangementmethod of reducing the virtual screen frame image 42 and arranging thereduced virtual screen frame image 42 in the margin (post-reductionarrangement) is to be included in the arrangement method for the virtualscreen frame image 42 (S104). Here, for example, an arrangement methodof reducing the virtual screen frame image 42 such that the size of thevirtual screen frame image 42 becomes same as the size of the margin andthen arranging the reduced virtual screen frame image 42 in the marginmay be determined to be included in the arrangement method for thevirtual screen frame image 42. In this case, a manifest file indicatingthe size of the virtual screen frame image 42 after reduced isgenerated.

FIG. 12 depicts a region R4 corresponding to the virtual screen frameimage 42. Thus, in the following description, the horizontal length ofthe virtual screen frame image 42 is represented by L3 and the verticallength of the virtual screen frame image 42 is represented by L4 asdepicted in FIG. 12. It is to be noted that, in the case where it isdecided by the process indicated at S104 that the method of reducing andarranging is to be included in the arrangement method for the virtualscreen frame image 42, the horizontal length of the virtual screen frameimage 42 after reduced is represented by L3 and the vertical length isrepresented by L4.

In the case where it is decided by the process indicated at S103 thatthe size of the margin is equal to or greater than the size of thevirtual screen frame image 42 (S103: Y) or in the case where the processindicated at S104 ends, the arrangement determination section 64 decideswhether or not the length L1 is equal to or greater than the length L2(Sl05). Here, that the length L1 is equal to or greater than the lengthL2 is equivalent to that a right portion with respect to the VR frameimage 40 has more room than a lower portion with respect to the VR frameimage 40. Conversely, that the length L1 is smaller than the length L2is equivalent to that a lower portion with respect to the VR frame image40 has more room than a right portion with respect to the VR frame image40.

In the case where it is decided that the length L is equal to or greaterthan the length L2 (S105: Y), the arrangement determination section 64decides whether or not the length L1 is equal to or greater than thelength L3 (S106). That the length LU is equal to or greater than thelength L3 is equivalent to that a virtual screen frame image 42 can bearranged in the same direction as that of the VR frame image 40 in aright portion with respect to the VR frame image 40. Conversely, thatthe length L1 is not equal to or greater than the length 13 signifiesthat, in the case where the virtual screen frame image 42 is disposedside by side in the same direction as that of the VR frame image 40 inthe margin, the virtual screen frame image 42 protrudes from the margin.

It is assumed here that it is decided that the length L1 is equal to orgreater than the length L3 (S106: Y). In this case, the arrangementdetermination section 64 decides that to arrange the virtual screenframe image 42 in a direction same as that of the VR frame image 40 inthe right portion with respect to the VR frame image 40 (normalarrangement (left-right)) is to be included in the arrangement methodfor the virtual screen frame image 42 (S107). Then, the processingindicated by the present example of processing is ended. In this case, amanifest file indicating that divisional images of the virtual screenframe image 42 are arranged side by side horizontally in a directionsame as that of the VR frame image 40 is generated.

In the case where it is decided by the process indicated at S106 thatthe length L1 is smaller than the length L3 (S106: N), the arrangementdetermination section 64 decides whether or not the length L1 is equalto or greater than the length L4 (S108). That the length L1 is equal toor greater than the length L4 is equivalent to that it is possible toarrange the virtual screen frame image 42 in a direction different fromthat of the VR frame image 40 in the margin.

It is assumed here that the length L1 is equal to or greater than thelength L4 (S108: Y). In this case, the arrangement determination section64 decides that to arrange the virtual screen frame image 42 in adirection different from that of the VR frame image 40 in the margin isto be included in the arrangement method for the virtual screen frameimage 42 (S109). Here, for example, it is determined that to arrange thevirtual screen frame image 42 in a direction rotated by 90 degrees withrespect to the VR frame image 40 in the right portion with respect tothe VR frame image 40 (rotational arrangement (left-right)) is to beincluded in the arrangement method for the virtual screen frame image42. Then, the processing indicated by the present example of processingis ended. It is to be noted that, it may be determined that the virtualscreen frame image 42 is to be arranged in a direction rotated by 90degrees clockwise with respect to the VR frame image 40, or it may bedetermined that the virtual screen frame image 42 is to be arranged in adirection rotated by 90 degrees counterclockwise. In this case, amanifest file indicating that divisional images of the virtual screenframe image 42 are arranged side by side horizontally in a directiondifferent from that of the VR frame image 40 is generated.

In the case where it is decided in the process indicated at S108 thatthe length L1 is smaller than the length L4 (S108: N), the arrangementdetermination section 64 decides whether or not it is possible toarrange the virtual screen frame image 42 in the margin by changing thedirection in which the divisional images of the virtual screen frameimage 42 are lined up (S110). Here, for example, it may be confirmedwhether or not it is possible to place the virtual screen frame image 42in the margin by changing the direction in which, where the virtualscreen frame image 42 is divided vertically into n, the n divisionalimages of the virtual screen frame image 42 are lined up from thehorizontal direction to the vertical direction successively changing theinteger n in order from 1 to a predetermined number N. Then, in the casewhere it is confirmed that the virtual screen frame image 42 fits in themargin at a certain integer n, it may be decided that the virtual screenframe image 42 can be arranged if it is divided. On the other hand, inthe case where it is decided that the virtual screen frame image 42 doesnot fit in the margin at any integer n from 1 to the predeterminednumber N, it may be decided that the virtual screen frame image 42cannot be arranged even if it is divided.

It is assumed that it is possible to arrange the virtual screen frameimage 42 in the margin by changing the lined up direction of thedivisional images of the virtual screen frame image 42 (S110: Y). Inthis case, the arrangement determination section 64 determines that toarrange the virtual screen frame image 42 in the margin by changing thelined up direction of the divisional images of the virtual screen frameimage 42 is to be included in the arrangement method for the virtualscreen frame image 42 (S111). Here, for example, it is decided that tochange the direction in which, where the virtual screen frame image 42is divided into n vertically, the n divisional images are lined up fromthe horizontal direction to the vertical direction and arrange the ndivisional images on the right with respect to the VR frame image 40(divisional arrangement (left-right)) is to be included in thearrangement method for the virtual screen frame image 42. Then, theprocessing indicated by the present example of processing is ended. Inthis case, a manifest file indicating that the divisional images of thevirtual screen frame image 42, which are arranged side by sidevertically, are arranged on the right with respect to the VR frame image40 is generated.

On the other hand, it is assumed that it is not possible to arrange thevirtual screen frame image 42 in the margin even if the lined updirection of the divisional images of the virtual screen frame image 42is changed (S110: N). In this case, the arrangement determinationsection 64 performs notification of suggesting a person in charge ofvideo delivery to determine an arrangement method (S112). Here, thenotification to a person in charge of video delivery may be performed,for example, by display outputting or sound outputting. Then, theprocessing indicated by the present example of processing is ended. Inthis case, for example, it is sufficient if a value of an attributerepresentative of the arrangement method for the virtual screen frameimage 42 is set in the manifest file by operation of the operationsection 12 f by a person in charge of video delivery.

In the case where it is decided by the process indicated at S105 thatthe length L1 is smaller than the length L2 (S105: N), the arrangementdetermination section 64 decides whether or not the length L2 is equalto or greater than the length L4 (S113). That the length L2 is equal toor greater than the length L4 is equivalent to that the virtual screenframe image 42 can be arranged in a direction same as that of the VRframe image 40 under the VR frame image 40. Conversely, that the lengthL2 is not equal to or greater than the length L4 is equivalent to that,when the divisional images of the virtual screen frame image 42 arearranged lined up in a direction same as that of the VR frame image 40in the margin, the virtual screen frame image 42 protrudes from themargin.

Here, it is assumed that the length L2 is equal to or greater than thelength L4 (S113: Y). In this case, the arrangement determination section64 determines that to arrange the virtual screen frame image 42 in adirection same as that of the VR frame image 40 under the VR frame image40 (normal arrangement (up-down)) is to be included in the arrangementmethod for the virtual screen frame image 42 (S114). Then, theprocessing indicated by the present example of processing is ended. Inthis case, a manifest file in which the divisional images of the virtualscreen frame image 42 are arranged lined up horizontally in a directionsame as that of the VR frame image 40 is generated.

In the case where it is decided by the process indicated at S113 thatthe length L2 is smaller than the length L4 (S113: N), the arrangementdetermination section 64 decides whether or not the length L2 is equalto or greater than the length L3 (S115). That the length L2 is equal toor greater than the length L3 is equivalent to that it is possible toarrange the virtual screen frame image 42 in a direction different fromthat of the VR frame image 40 in the margin.

Here, it is assumed that the length L2 is equal to or greater than thelength L3 (S115: Y). In this case, the arrangement determination section64 determines that to arrange the virtual screen frame image 42 in adirection different from that of the VR frame image 40 in the margin isto be included in the arrangement method for the virtual screen frameimage 42 (S116). Here, for example, it is determined that to arrange thevirtual screen frame image 42 in a direction rotated by 90 degrees withrespect to the VR frame image 40 under the VR frame image 40 (rotationalarrangement (up-down)) is to be included in the arrangement method forthe virtual screen frame image 42. Then, the processing indicated by thepresent example of processing is ended. It is to be noted that, it maybe determined that the virtual screen frame image 42 is to be arrangedin a direction rotated by 90 degrees clockwise with respect to the VRframe image 40, or it may be determined that the virtual screen frameimage 42 is to be arranged in a direction rotated by 90 degreescounterclockwise. In this case, a manifest file is generated in which itis indicated that the divisional images of the virtual screen frameimage 42 are arranged lined up vertically in a direction different fromthat of the VR frame image 40.

In the case where it is decided by the process indicated at S115 thatthe length L2 is smaller than the length L3 (S115: N), the arrangementdetermination section 64 decides whether or not it is possible toarrange the virtual screen frame image 42 in the margin by changing thedirection in which the divisional images of the divided virtual screenframe image 42 are lined up (S117). Here, for example, it may beconfirmed whether or not it is possible to place the virtual screenframe image 42 in the margin by changing the direction in which, wherethe virtual screen frame image 42 is divided horizontally into n, the ndivisional images of the virtual screen frame image 42 are lined up fromthe vertical direction to the horizontal direction successively changingthe integer n in order from 1 to a predetermined number N. Then, in thecase where it is confirmed that the virtual screen frame image 42 fitsin the margin at a certain integer n, it may be decided that the virtualscreen frame image 42 can be arranged if it is divided. On the otherhand, in the case where it is decided that the virtual screen frameimage 42 does not fit in the margin at any integer n from 1 to thepredetermined number N, it may be decided that the virtual screen frameimage 42 cannot be arranged even if it is divided.

It is assumed that it is possible to arrange the virtual screen frameimage 42 in the margin by changing the lined up direction of thedivisional images of the virtual screen frame image 42 (S117: Y). Inthis case, the arrangement determination section 64 determines that toarrange the virtual screen frame image 42 in the margin by changing thelined up direction of the divisional images of the virtual screen frameimage 42 is to be included in the arrangement method for the virtualscreen frame image 42 (S118). Here, for example, it is decided that tochange the direction in which, where the virtual screen frame image 42is divided into n horizontally, the n divisional images are lined upfrom the vertical direction to the horizontal direction and arrange then divisional images under the VR frame image 40 (divisional arrangement(up-down)) is to be included in the arrangement method for the virtualscreen frame image 42.

Then, the processing indicated by the present example of processing isended. In this case, a manifest file indicating that the divisionalimages of the virtual screen frame image 42, which are arranged side byside horizontally, are arranged under the VR frame image 40 isgenerated.

On the other hand, it is assumed that it is not possible to arrange thevirtual screen frame image 42 in the margin even if the lined updirection of the divisional images of the virtual screen frame image 42is changed (S117: N). In this case, the arrangement determinationsection 64 performs notification of suggesting a person in charge ofvideo delivery to determine an arrangement method (S119). Then, theprocessing indicated by the present example of processing is ended. Inthis case, for example, it is sufficient if a value of an attributerepresentative of the arrangement method for the virtual screen frameimage 42 is set in the manifest file by operation of the operationsection 12 f by a person in charge of video delivery.

For example, it is assumed that the horizontal length of the syntheticframe image is 3840 pixels and the vertical length is 1920 pixels.Further, it is assumed that the horizontal length of the VR frame image40 is 2760 pixels and the vertical length is 1920 pixels. Further, it isassumed that the horizontal length of the virtual screen frame image 42is 1920 pixels and the vertical length is 1080 pixels.

In this case, the size of the margin is 2073600 square pixels and isequal to the size of the virtual screen frame image 42 (S103: Y).

Then, in this case, the length L1 is 1080 pixels and the length L2 is 0pixel. Further, the length L3 is 1920 pixels and the length L4 is 1080pixels.

In the present example, the length L1 is equal to or greater than thelength L2 (S105: Y); the length L1 is smaller than the length L3 (S105:N); and the length L1 is equal to or greater than the length L4 (S108:Y). Therefore, it is determined that to arrange the virtual screen frameimage 42 in a direction rotated by 90 degrees with respect to the VRframe image 40 on the right with respect to the VR frame image 40(rotational arrangement (left-right)) is to be included in thearrangement method for the virtual screen frame image 42 (S110).

Further, it is assumed that, for example, the horizontal length of thesynthetic frame image is 3840 pixels and the vertical length is 2160pixels. Then, it is assumed that the horizontal length of the VR frameimage 40 is 3840 pixels and the vertical length is 1920 pixels. Further,it is assumed that the horizontal length of the virtual screen frameimage 42 is 1920 pixels and the vertical length is 1080 pixels.

In this case, the size of the margin is 921600 square pixels and issmaller than the size of the virtual screen frame image 42 (S103: N).Therefore, in this case, it is determined that the post-reductionarrangement is to be included in the arrangement method for the virtualscreen frame image 42 (S104). Here, for example, it may be determinedthat to reduce the size of the virtual screen frame image 42 so as to besame as the size of the margin without changing the aspect ratio is tobe included in the arrangement method for the virtual screen frame image42. For example, it may be determined that to reduce the horizontallength of the virtual screen frame image 42 to 1280 pixels and reducethe vertical length to 720 pixels is to be included in the arrangementmethod for the virtual screen frame image 42.

In this case, the length L1 is 0 pixel, and the length L2 is 240 pixels.Further, the length L3 is 1280 pixels, and the length L4 is 720 pixels.

In the present example, the length L1 is smaller than the length L2(S105: N); the length L2 is smaller than the length L4 (S112: N); andthe length L2 is smaller than the length L3 (S114: N). Then, in thiscase, if the virtual screen frame image 42 is divided into three in thehorizontal direction and the three divisional images are arranged sideby side horizontally, then they fit in the margin (S115: Y). Therefore,it is determined that the divisional arrangement (up-down) is to beincluded in the arrangement method for the virtual screen frame image 42(S116).

It is to be noted that, in the foregoing description, the timing atwhich a manifest file is to be generated does not matter in particular.For example, generation and transmission of a manifest file may beperformed before live relay is started, or, for example, data indicativeof values of attributes of individual frame images may be transmitted inassociation with the synthetic frame image from the video deliveryapparatus 12 to the user system 14. For example, if this is done, theneven during live relay, values of attributes of individual frame imagescan be changed smoothly.

Now, an example of a flow of processing performed by the HMD 20according to the present embodiment for causing an image extracted froma decoded synthetic frame image to be displayed is described withreference to a flow chart exemplified in FIG. 13.

First, the image extraction section 86 specifies a region occupied bythe VR frame image 40 and a region occupied by the virtual screen frameimage 42 in a decoded synthetic frame image (S201). Here, for example,the regions mentioned may be specified on the basis of values ofattributes indicated by the manifest file stored in the manifest filestorage section 82.

Then, the image extraction section 86 extracts the VR frame image 40arranged in the region occupied by the VR frame image 40 and specifiedby the process indicated at S201 (S202).

Then, the image extraction section 86 extracts the virtual screen frameimage 42 arranged in the region occupied by the virtual screen frameimage 42 and specified by the process indicated at S201 (S203). Here,for example, it is sometimes indicated in the manifest file that thevirtual screen frame image 42 is arranged rotated. In this case, aprocess for reproducing the virtual screen frame image 42 beforerotation by rotating the virtual screen frame image 42 after rotation isperformed by the process indicated at S203. Further, for example, it issometimes indicated in the manifest file that the virtual screen frameimage 42 is arranged divided. In this case, by the process indicated atS203, a process of reproducing the virtual screen frame image 42 beforedivision by re-arranging the divisional images of the virtual screenframe image 42 after division is performed.

Then, the display controlling section 90 sets a value of the viewpointposition data, a value of the gaze direction data, and a value of thedisplay control flag retained therein on the basis of an operationaccepted by the display controlling operation acceptance section 88 in aloop in the current operation cycle (S204). Here, for example, when apredetermined operation for the controller 30 is accepted, the value ofthe display control flag may be updated from 1 to 0 or from 0 to 1.

Then, the display controlling section 90 generates and maps a textureimage 44 to the inner side of the background object 48 (S205). Here, thetexture image 44 is generated, for example, on the basis of the VR frameimage 40 extracted by the process indicated at S202, the virtual screenframe image 42 extracted by the process indicated at S203, and the valueof the display control flag set by the process indicated at S204. Here,for example, a position or a size of the virtual screen frame image 42with respect to the VR frame image 40 may be determined, for example, onthe basis of values of attributes indicated by the manifest file. Then,a texture image 44 in which the virtual screen frame image 42 isarranged at the determined position and in the determined size may begenerated.

Further, for example, in the case where the value of the display controlflag is 0, such a texture image 44 a that does not include the virtualscreen frame image 42 as depicted in FIG. 5A may be generated. On theother hand, for example, in the case where the value of the displaycontrol flag is 1, such a texture image 44 b in which the virtual screenframe image 42 is arranged superimposed on the VR frame image 40 asdepicted in FIG. 5B may be generated. In this case, the virtual screenframe image 42 may be arranged superimposed on the VR frame image 40after it is deformed so as to be displayed as a rectangular image on thedisplay section 20 f as described hereinabove. Further, the virtualscreen frame image 42 may be mapped as a texture image 44 b on aplate-shaped virtual object disposed in the virtual space 46 asdescribed hereinabove. In this manner, the virtual screen frame image 42may be arranged as the texture image 44 b independent of the VR frameimage 40 in the virtual space 46.

Then, the display controlling section 90 controls the display section 20f to display an image representative of a state in which a gazedirection 52 indicated by a value of the gaze direction data set by theprocess indicated at S204 is viewed from a viewpoint 50 arranged at aposition indicated by the value of the viewpoint position data set bythe process indicated at S204 (S206).

Then, the processing returns to the process indicated at S201, and theprocesses beginning with S201 are executed for a synthetic frame imageof a next frame.

It is to be noted that the arrangement method of the VR frame image 40and the virtual screen frame image 42 is not limited to those describedabove.

For example, FIG. 14 depicts an example of arrangement in which twoimages of a virtual screen frame image 42(1) and another virtual screenframe image 42(2) are arranged side by side vertically on the right withrespect to the VR frame image 40. Further, FIG. 15 depicts a manifestfile corresponding to FIG. 14. In the manifest file depicted in FIG. 15,values of attributes of the VR frame image 40 are indicated in linesfrom a line next to [VR_Movie_1] to a line preceding to[VirtualScreen_1]. Further, values of attributes of the virtual screenframe image 42(1) are indicated in lines from a line next to[VirtualScreen_1] to a line preceding to [VirtualScreen_2]. Furthermore,values of attributes of the virtual screen frame image 42(2) areindicated in lines from a line next to [VirtualScreen_2].

Further, for example, FIG. 16 depicts an example of arrangement in whichdivisional images of the virtual screen frame image 42 divided intothree are arranged side by side horizontally under a VR frame image40(1) and another VR frame image 40(2) arranged side by sidehorizontally. Further, FIG. 17 depicts a manifest file corresponding toFIG. 16. In the manifest file depicted in FIG. 17, values of attributesof the VR frame image 40(1) are indicated in lines from a line next to[VR_Movie_1] to a line preceding to [VR_Movie_2]. Further, values ofattributes of the VR frame image 40(2) are indicated in lines from aline next to [VR_Movie_2] to a line preceding to [VirtualScreen_1].Further, values of attributes of the virtual screen frame image 42 areindicated from a line next to [VirtualScreen_1].

As depicted in FIGS. 14 and 16, a synthetic frame image may be generatedby synthesizing individual frame images captured by three or morecameras 16. For example, each of two cameras 16 may individuallygenerate a VR frame image 40. In this case, it may be switchable whetherthe VR frame image 40(1) is to be displayed or the VR frame image 40(2)is to be displayed, for example, by the controller 30 operated by theuser. Further, for example, each of two cameras 16 may individuallygenerate a virtual screen frame image 42. In this case, it may beswitchable whether the virtual screen frame image 42(1) is to bedisplayed or the virtual screen frame image 42(2) is to be displayed,for example, by the controller 30 operated by the user.

Further, for example, in FIG. 18, an example of arrangement is depictedin which divisional images of the virtual screen frame image 42 dividedinto three are arranged side by side horizontally under a VR frame image40 (for the left eye) and another VR frame image 40 (for the right eye),which form a three-dimensional image. In the example of FIG. 18, the VRframe image 40 that is a three-dimensional image is arranged in the sideby side form. Further, FIG. 19 depicts a manifest file corresponding toFIG. 18. In the manifest file depicted in FIG. 19, values of attributesof the VR frame image 40 (for the left eye) and the VR frame image 40(for the right eye) are indicated in lines from a line next to[VR_Movie_1] to a line preceding to [VirtualScreen_1]. Further, valuesof attributes of the virtual screen frame image 42 are indicated inlines from a line next to [VirtualScreen_1].

A three-dimensional image may be used as the VR frame image 40 asdepicted in FIG. 18. In this case, for example, the camera 16 a may be astereo camera.

It is to be noted that the manifest file may include a uniform resourcelocator (URL) of a video of a delivery target. Further, for example,when the user executes a video displaying application installed in theuser system 14, the user system 14 may access the URL included in themanifest file. Then, through the access to the URL in this manner, avideo may be delivered from the video delivery apparatus 12 to the usersystem 14.

It is to be noted that the present invention is not limited to theembodiment described above.

For example, the user system 14 is not limited, for example, to thatdepicted in FIG. 3 and may be, for example, a personal computer or thelike.

Further, the particular character strings and numerical values describedabove and the particular character strings and numerical values in thedrawings are exemplary and the present invention is not restricted tosuch character strings or numerical values.

The invention claimed is:
 1. An image arrangement determinationapparatus comprising: an acquisition section configured to acquire aplurality of images including a first image and a second image having anaspect ratio different from that of the first image; a marginspecification section configured to specify a margin when the firstimage is arranged by a given first arrangement method in a regionoccupied by a third image that is a delivery target; and an arrangementdetermination section configured to determine if a size of the margin isequal to or greater than a size of the second image, wherein, if thesize of the margin is greater than the size of the second image, thearrangement determination section arranges the second image in themargin adjacent the first image without dividing the second image, andwherein, if the size of the margin is less than the size of the secondimage, the arrangement determination section divides the second imageinto a plurality of divisional images and determines a secondarrangement method from a plurality of second arrangement methods bywhich the plurality of divisional images are arranged in the margin ofthe third image adjacent the first image based on a shape of the marginand a shape of the second image.
 2. The image arrangement determinationapparatus according to claim 1, further comprising: a transmissionsection configured to transmit the third image in which the first imageis arranged by the first arrangement method and the second image orplurality of divisional images are arranged by the second arrangementmethod.
 3. The image arrangement determination apparatus according toclaim 1, wherein when the size of the margin is greater than the size ofthe second image, and, in a case in which it is decided that, when thesecond image is arranged lined up with the first image in a samedirection as that of the first image in the margin, the second imageprotrudes from the margin, the determination section rotates the secondimage before arranging the second image in the margin adjacent thefirst.
 4. A display controlling apparatus comprising: an image receptionsection configured to receive a third image that includes a first regionoccupied by a first image and a second region occupied by a plurality ofdivisional images, wherein the plurality of divisional images form asecond image when combined, the second image having an aspect ratiodifferent from that of the first image; a data reception sectionconfigured to receive data indicative of a first arrangement method thatis an arrangement method of the first image in the third image and asecond image arrangement method that is an arrangement method of theplurality of divisional images in the third image; an extraction sectionconfigured to extract the first image and the plurality of divisionalimages from the third image based on the first arrangement method andthe second arrangement method indicated by the data and to combine theplurality of divisional images to form the second image; and a displaycontrolling section configured to cause at least one of the first imageand the second image extracted to be displayed.
 5. An image arrangementdetermination method comprising: acquiring a plurality of imagesincluding a first image and a second image having an aspect ratiodifferent from that of the first image; specifying a margin when thefirst image is arranged by a given first arrangement method in a regionoccupied by a third image that is a delivery target; determining if asize of the margin is equal to or greater than a size of the secondimage, wherein, if the size of the margin is greater than the size ofthe second image, arranging the second image in the margin adjacent thefirst image without dividing the second image, and wherein, if themargin has a size smaller than a size of the second image, determiningan arrangement method by which the second image is reduced in size andthen subsequently arranged in the margin as the second arrangementmethod.
 6. A display controlling method comprising: receiving a thirdimage that includes a first region occupied by a plurality of divisionalimages, wherein the plurality of divisional images form a second imagewhen combined; receiving data indicative of a first arrangement methodthat is an arrangement method of the first image in the third image anda second image arrangement method that is an arrangement method of theplurality of divisional images in the third image; extracting the firstimage and the plurality of divisional images from the third image basedon the first arrangement method and the second arrangement methodindicated by the data; combining the plurality of divisional images toform the second image; and displaying the first image and the secondimage.